To warrant clinical adoption AI models require a multi-faceted implementation evaluation.

Despite artificial intelligence (AI) technology progresses at unprecedented rate, our ability to translate these advancements into clinical value and adoption at the bedside remains comparatively limited. This paper reviews the current use of implementation outcomes in randomized controlled trials evaluating AI-based clinical decision support and found limited adoption. To advance trust and clinical adoption of AI, there is a need to bridge the gap between traditional quantitative metrics and implementation outcomes to better grasp the reasons behind the success or failure of AI systems and improve their translation into clinical value.

Cognitive and psychological recovery patterns across different care pathways 12 months after hospitalization for COVID-19: A multicenter cohort study (CO-FLOW).

BACKGROUND: The comparison of recovery patterns for different care pathways following COVID-19 is necessary for optimizing rehabilitation strategies. OBJECTIVES: To evaluate cognitive and psychological outcomes across different care pathways up to 12 months after hospitalization for COVID-19. METHODS: CO-FLOW is an ongoing multicenter prospective cohort study with assessments at 3, 6, and 12 months after hospitalization for COVID-19. The main outcomes are cognitive deficits (Montreal Cognitive Assessment, score <26), cognitive failure (Cognitive Failure Questionnaire, score >43), posttraumatic stress disorder (PTSD; Impact of Event Scale-Revised, score ≥33), and anxiety and depression (Hospital Anxiety and Depression Scale, subscale score ≥11). RESULTS: In total, data from 617 participants were analyzed. Mean age was 59.7 (SD 11.4) years and 188 (31%) were female. Significant recovery occurred within the first 6 months post-discharge (p ≤ 0.001). Cognitive deficits persisted in 21% (101/474), and psychological problems in 15% (74/482) of people at 12 months. Significantly improved cognition scores were reported for people who did not receive rehabilitation ('No-rehab'; 124/617, 20%; mean difference, MD 2.32, 95% CI 1.47 to 3.17; p<0.001), those who received community-based rehabilitation ('Com-rehab'; 327/617, 53%; MD 1.27, 95% CI 0.77 to 1.78; p<0.001), and those who received medical rehabilitation ('Med-rehab'; 86/617, 14%; MD 1.63, 95% CI 0.17 to 3.10; p = 0.029). Med-rehab participants experienced more cognitive failure from 3 to 6 months (MD 4.24, 95% 1.63 to 6.84; p = 0.001). Com-rehab showed recovery for PTSD (MD -2.43, 95% -3.50 to -1.37; p<0.001), anxiety (MD -0.67, 95% -1.02 to -0.32; p<0.001), and depression (MD -0.60, 95% -0.96 to -0.25; p<0.001), but symptoms persisted at 12 months. CONCLUSIONS: Survivors of COVID-19 showed cognitive and psychological recovery, especially within the first 6 months after hospitalization. Most persistent problems were related to cognitive functioning at 12 months. Recovery differed rehabilitation settings. Additional cognitive or psychological support might be warranted in people who medical or community-based rehabilitation.

Symptoms persisting after hospitalisation for COVID-19: 12 months interim results of the CO-FLOW study.

Introduction: A large proportion of patients experience a wide range of sequelae after acute COVID-19, especially after severe illness. The long-term health sequelae need to be assessed. Our objective was to longitudinally assess persistence of symptoms and clusters of symptoms up to 12 months after hospitalisation for COVID-19 and to assess determinants of the main persistent symptoms. Methods: In this multicenter prospective cohort study patients with COVID-19 are followed up for 2 years with measurements at 3, 6, 12 and 24 months after hospital discharge. Here, we present interim results regarding persistent symptoms up to 12 months. Results: We included 492 patients; mean±sd age was 60.2±10.7 years, 335 (68.1%) were males, median length of hospital stay was 11 (6.0-27.0) days. At 3 months after discharge 97.0% of the patients had at least one persisting symptom, this declined to 95.5% and 92.0% at 6 and 12 months, respectively (p=0.010). Muscle weakness, exertional dyspnoea, fatigue, and memory and concentration problems were the most prevalent symptoms with rates over 50% during follow-up. Over time, muscle weakness, hair loss and exertional dyspnoea decreased significantly (p<0.001), while other symptoms such as fatigue, concentration and memory problems, anosmia and ageusia persisted. Symptoms from the physical and respiratory cluster declined significantly over time, in contrast to the fatigue and cognitive symptom clusters. Conclusion: The majority of patients experienced COVID-19 sequelae up to 12 months after severe infection. Whereas physical and respiratory symptoms showed slow gradual decline, fatigue and cognitive symptoms did not evidently resolve over time.

Effect of intensive care unit-specific virtual reality (ICU-VR) to improve psychological well-being in ICU survivors: study protocol for an international, multicentre, randomised controlled trial-the HORIZON-IC study.

INTRODUCTION: A substantial proportion of intensive care unit (ICU) survivors develop psychological impairments after ICU treatment, part of the postintensive care syndrome, resulting in a decreased quality of life. Recent data suggest that an ICU-specific virtual reality intervention (ICU-VR) for post-ICU patients is feasible and safe, improves satisfaction with ICU aftercare, and might improve psychological sequelae. In the present trial, we firstly aim to determine whether ICU-VR is effective in mitigating post-traumatic stress disorder (PTSD)-related symptoms and secondly to determine the optimal timing for initiation with ICU-VR. METHODS AND ANALYSIS: This international, multicentre, randomised controlled trial will be conducted in 10 hospitals. Between December 2021 and April 2023, we aim to include 300 patients who have been admitted to the ICU ≥72 hours and were mechanically ventilated ≥24 hours. Patients will be followed for 12 consecutive months. Patients will be randomised in a 1:1:1 ratio to the early ICU-VR group, the late ICU-VR group, or the usual care group. All patients will receive usual care, including a mandatory ICU follow-up clinic visit 3 months after ICU discharge. Patients in the early ICU-VR group will receive ICU-VR within 2 weeks after ICU discharge. Patients in the late VR group will receive ICU-VR during the post-ICU follow-up visit. The primary objective is to assess the effect of ICU-VR on PTSD-related symptoms. Secondary objectives are to determine optimal timing for ICU-VR, to assess the effects on anxiety-related and depression-related symptoms and health-related quality of life, and to assess patient satisfaction with ICU aftercare and perspectives on ICU-VR. ETHICS AND DISSEMINATION: The Medical Ethics Committee United, Nieuwegein, the Netherlands, approved this study and local approval was obtained from each participating centre (NL78555.100.21). Our findings will be disseminated by presentation of the results at (inter)national conferences and publication in scientific, peer-reviewed journals. TRIAL REGISTRATION NUMBER: NL9812.

Development and validation of an early warning model for hospitalized COVID-19 patients: a multi-center retrospective cohort study.

BACKGROUND: Timely identification of deteriorating COVID-19 patients is needed to guide changes in clinical management and admission to intensive care units (ICUs). There is significant concern that widely used Early warning scores (EWSs) underestimate illness severity in COVID-19 patients and therefore, we developed an early warning model specifically for COVID-19 patients. METHODS: We retrospectively collected electronic medical record data to extract predictors and used these to fit a random forest model. To simulate the situation in which the model would have been developed after the first and implemented during the second COVID-19 'wave' in the Netherlands, we performed a temporal validation by splitting all included patients into groups admitted before and after August 1, 2020. Furthermore, we propose a method for dynamic model updating to retain model performance over time. We evaluated model discrimination and calibration, performed a decision curve analysis, and quantified the importance of predictors using SHapley Additive exPlanations values. RESULTS: We included 3514 COVID-19 patient admissions from six Dutch hospitals between February 2020 and May 2021, and included a total of 18 predictors for model fitting. The model showed a higher discriminative performance in terms of partial area under the receiver operating characteristic curve (0.82 [0.80-0.84]) compared to the National early warning score (0.72 [0.69-0.74]) and the Modified early warning score (0.67 [0.65-0.69]), a greater net benefit over a range of clinically relevant model thresholds, and relatively good calibration (intercept = 0.03 [- 0.09 to 0.14], slope = 0.79 [0.73-0.86]). CONCLUSIONS: This study shows the potential benefit of moving from early warning models for the general inpatient population to models for specific patient groups. Further (independent) validation of the model is needed.

Physical recovery across care pathways up to 12 months after hospitalization for COVID-19: A multicenter prospective cohort study (CO-FLOW).

Backgroud: The sudden COVID-19 pandemic forced quick development of care pathways for patients with different needs. Trajectories of physical recovery in hospitalized patients for COVID-19 following different care pathways are unknown. We aimed to assess trajectories of physical recovery and levels of physical function reached within the different care pathways. Additionally, we assessed differences in physical function across care pathways at follow-up visits. Methods: This multicenter prospective cohort study of adults who had been hospitalized for COVID-19 was performed in 10 centers, including 7 hospitals (1 academic and 6 regional hospitals) and 3 rehabilitation centers (1 medical rehabilitation center and 2 skilled nursing facilities), located in the Netherlands. Study visits were performed at 3, 6, and 12 months post-hospital discharge and included assessment of cardiorespiratory fitness (6 min walk test [6MWT], 1 min sit-to-stand test [1MSTST]), muscle strength (maximum handgrip strength [HGS]) and mobility (de Morton Mobility Index [DEMMI]). Findings: We report findings for 582 patients who had been discharged from hospital between March 24, 2020 and June 17, 2021. Patients had a median age of 60·0 years, 68·9% (401/582) were male, 94·6% (561/582) had received oxygen therapy, and 35·2% (205/582) mechanical ventilation. We followed patients across four different rehabilitation settings: no rehabilitation (No-rehab, 19·6% [114/582]), community-based rehabilitation (Com-rehab, 54·1% [315/582]), medical rehabilitation (Med-rehab, 13·7% [80/582]), and rehabilitation in a skilled nursing facility (SNF-rehab, 12·5% [73/582]). Overall, outcomes in 6MWT (14·9 meters [95% CI 7·4 to 22·4]), 1MSTST (2·2 repetitions [1·5 to 2·8]), and HGS (3·5 kg [2·9 to 4·0]) improved significantly (p<0·001) from 3 to 6 months and only HGS from 6 to 12 months (2·5 kg [1·8 to 3·1]; p<0·001). DEMMI scores did not significantly improve over time. At 3 months, percentage of normative values reached in 1MSTST differed significantly (p<0.001) across care pathways, with largest impairments in Med- and SNF-rehab groups. At 12 months these differences were no longer significant, reaching, overall, 90·5% on 6MWD, 75·4% on 1MSTST, and 106·9% on HGS. Interpretation: Overall, physical function improved after hospitalization for COVID-19, with largest improvement within 6 months post-discharge. Patients with rehabilitation after hospital discharge improved in more than one component of physical function, whereas patients without rehabilitation improved solely in muscle strength. Patients who received rehabilitation, and particularly patients with Med- and SNF-rehab, had more severe impairment in physical function at 3 months, but reached equal levels at 12 months compared to patients without follow-up treatment. Our findings indicate the importance of rehabilitation. Funding: ZonMw, Rijndam Rehabilitation, Laurens (The Netherlands).

Optimizing discharge after major surgery using an artificial intelligence-based decision support tool (DESIRE): An external validation study.

BACKGROUND: In the DESIRE study (Discharge aftEr Surgery usIng aRtificial intElligence), we have previously developed and validated a machine learning concept in 1,677 gastrointestinal and oncology surgery patients that can predict safe hospital discharge after the second postoperative day. Despite strong model performance (area under the receiver operating characteristics curve of 0.88) in an academic surgical population, it remains unknown whether these findings can be translated to other hospitals and surgical populations. We therefore aimed to determine the generalizability of the previously developed machine learning concept. METHODS: We externally validated the machine learning concept in gastrointestinal and oncology surgery patients admitted to 3 nonacademic hospitals in The Netherlands between January 2017 and June 2021, who remained admitted 2 days after surgery. Primary outcome was the ability to predict hospital interventions after the second postoperative day, which were defined as unplanned reoperations, radiological interventions, and/or intravenous antibiotics administration. Four forest models were locally trained and evaluated with respect to area under the receiver operating characteristics curve, sensitivity, specificity, positive predictive value, and negative predictive value. RESULTS: All models were trained on 1,693 epsiodes, of which 731 (29.9%) required a hospital intervention and demonstrated strong performance (area under the receiver operating characteristics curve only varied 4%). The best model achieved an area under the receiver operating characteristics curve of 0.83 (95% confidence interval [0.81-0.85]), sensitivity of 77.9% (0.67-0.87), specificity of 79.2% (0.72-0.85), positive predictive value of 61.6% (0.54-0.69), and negative predictive value of 89.3% (0.85-0.93). CONCLUSION: This study showed that a previously developed machine learning concept can predict safe discharge in different surgical populations and hospital settings (academic versus nonacademic) by training a model on local patient data. Given its high accuracy, integration of the machine learning concept into the clinical workflow could expedite surgical discharge and aid hospitals in addressing capacity challenges by reducing avoidable bed-days.

Developing, implementing and governing artificial intelligence in medicine: a step-by-step approach to prevent an artificial intelligence winter.

OBJECTIVE: Although the role of artificial intelligence (AI) in medicine is increasingly studied, most patients do not benefit because the majority of AI models remain in the testing and prototyping environment. The development and implementation trajectory of clinical AI models are complex and a structured overview is missing. We therefore propose a step-by-step overview to enhance clinicians' understanding and to promote quality of medical AI research. METHODS: We summarised key elements (such as current guidelines, challenges, regulatory documents and good practices) that are needed to develop and safely implement AI in medicine. CONCLUSION: This overview complements other frameworks in a way that it is accessible to stakeholders without prior AI knowledge and as such provides a step-by-step approach incorporating all the key elements and current guidelines that are essential for implementation, and can thereby help to move AI from bytes to bedside.

Intensive Care Unit-Specific Virtual Reality for Critically Ill Patients With COVID-19: Multicenter Randomized Controlled Trial.

BACKGROUND: Although psychological sequelae after intensive care unit (ICU) treatment are considered quite intrusive, robustly effective interventions to treat or prevent these long-term sequelae are lacking. Recently, it was demonstrated that ICU-specific virtual reality (ICU-VR) is a feasible and acceptable intervention with potential mental health benefits. However, its effect on mental health and ICU aftercare in COVID-19 ICU survivors is unknown. OBJECTIVE: This study aimed to explore the effects of ICU-VR on mental health and on patients' perceived quality of, satisfaction with, and rating of ICU aftercare among COVID-19 ICU survivors. METHODS: This was a multicenter randomized controlled trial. Patients were randomized to either the ICU-VR (intervention) or the control group. All patients were invited to an COVID-19 post-ICU follow-up clinic 3 months after hospital discharge, during which patients in the intervention group received ICU-VR. One month and 3 months later (4 and 6 months after hospital discharge), mental health, quality of life, perceived quality, satisfaction with, and rating of ICU aftercare were scored using questionnaires. RESULTS: Eighty-nine patients (median age 58 years; 63 males, 70%) were included. The prevalence and severity of psychological distress were limited throughout follow-up, and no differences in psychological distress or quality of life were observed between the groups. ICU-VR improved satisfaction with (mean score 8.7, SD 1.6 vs 7.6, SD 1.6 [ICU-VR vs control]; t64=-2.82, P=.006) and overall rating of ICU aftercare (mean overall rating of aftercare 8.9, SD 0.9 vs 7.8, SD 1.7 [ICU-VR vs control]; t64=-3.25; P=.002) compared to controls. ICU-VR added to the quality of ICU aftercare according to 81% of the patients, and all patients would recommend ICU-VR to other ICU survivors. CONCLUSIONS: ICU-VR is a feasible and acceptable innovative method to improve satisfaction with and rating of ICU aftercare and adds to its perceived quality. We observed a low prevalence of psychological distress after ICU treatment for COVID-19, and ICU-VR did not improve psychological recovery or quality of life. Future research is needed to confirm our results in other critical illness survivors to potentially facilitate ICU-VR's widespread availability and application during follow-up. TRIAL REGISTRATION: Netherlands Trial Register NL8835; https://www.trialregister.nl/trial/8835. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.1186/s13063-021-05271-z.

Persistent Health Problems beyond Pulmonary Recovery up to 6 Months after Hospitalization for COVID-19: A Longitudinal Study of Respiratory, Physical, and Psychological Outcomes.

Rationale: Data on longitudinal recovery after hospitalization for coronavirus disease (COVID-19) currently remain scarce, just as outcomes beyond 3 months of follow-up do. Objectives: To evaluate the sequelae up to 6 months after hospitalization for COVID-19 by considering 1) recovery as it relates to pulmonary function, radiological abnormalities, physical and mental health status, and health-related quality of life (HR-QoL) and 2) the predictors of the most clinically relevant sequelae. Methods: Patients were evaluated at 6 weeks, 3 months, and 6 months after hospitalization by using pulmonary function testing, radiological evaluation, and online questionnaires on the physical and mental health status and HR-QoL. Outcomes were analyzed using repeated-measurement analyses. Results: Ninety-two patients were included (mean age, 58.2 ± 12.3 yr; 58 [63.0%] men). The estimated percentage of patients with impaired forced vital capacity improved from 25% at 6 weeks to 11% at 6 months; for impaired diffusion capacity, this percentage improved from 63% to 46%. Radiologically, ground-glass opacity decreased but fibrosis persisted. The majority of patients (89.1%) still reported one or more symptoms 6 months after discharge. Fatigue decreased significantly over time (P = 0.006). Nonetheless, fatigue remained in 51% of the patients at 6 months. HR-QoL (nearly) normalized in most domains at 6 months, except for physical role functioning, with persistent fatigue and the length of hospitalization being the most important predictors. Conclusions: During the first 6 months after hospitalization for COVID-19, most patients demonstrated continuing recovery across all health domains, but persistent sequelae were frequent. Fatigue was the most frequent residual and persistent symptom up to 6 months after hospitalization, importantly impacting HR-QoL.

Virtual reality for relatives of ICU patients to improve psychological sequelae: study protocol for a multicentre, randomised controlled trial.

INTRODUCTION: Intensive care unit (ICU) admission of a relative might lead to psychological distress and complicated grief (post-intensive care syndrome-family; PICS-F). Evidence suggests that increased distress during ICU stay increases risk of PICS-F, resulting in difficulty returning to their normal lives after the ICU experience. Effective interventions to improve PICS-F are currently lacking. In the present trial, we hypothesised that information provision using ICU-specific Virtual Reality for Family members/relatives (ICU-VR-F) may improve understanding of the ICU and subsequently improve psychological well-being and quality of life in relatives of patients admitted to the ICU. METHODS AND ANALYSIS: This multicentre, clustered randomised controlled trial will be conducted from January to December 2021 in the mixed medical-surgical ICUs of four hospitals in Rotterdam, the Netherlands. We aim to include adult relatives of 160 ICU patients with an expected ICU length of stay over 72 hours. Participants will be randomised clustered per patient in a 1:1 ratio to either the intervention or control group. Participants allocated to the intervention group will receive ICU-VR-F, an information video that can be watched in VR, while the control group will receive usual care. Initiation of ICU-VR-F will be during their hospital visit unless participants cannot visit the hospital due to COVID-19 regulations, then VR can be watched digitally at home. The primary objective is to study the effect of ICU-VR-F on psychological well-being and quality of life up to 6 months after the patients' ICU discharge. The secondary outcome is the degree of understanding of ICU treatment and ICU modalities. ETHICS AND DISSEMINATION: The Medical Ethics Committee of the Erasmus Medical Centre, Rotterdam, the Netherlands, approved the study and local approval was obtained from each participating centre (NL73670.078.20). Our findings will be disseminated by presentation of the results at (inter)national conferences and publication in scientific, peer-reviewed journals. TRIAL REGISTRATION NUMBER: Netherlands Trial Register (TrialRegister.nl, NL9220).

Generating insights in uncharted territories: real-time learning from data in critically ill patients-an implementer report.

Introduction In the current situation, clinical patient data are often siloed in multiple hospital information systems. Especially in the intensive care unit (ICU), large volumes of clinical data are routinely collected through continuous patient monitoring. Although these data often contain useful information for clinical decision making, they are not frequently used to improve quality of care. During, but also after, pressing times, data-driven methods can be used to mine treatment patterns from clinical data to determine the best treatment options from a hospitals own clinical data.Methods In this implementer report, we describe how we implemented a data infrastructure that enabled us to learn in real time from consecutive COVID-19 ICU admissions. In addition, we explain our step-by-step multidisciplinary approach to establish such a data infrastructure.Conclusion By sharing our steps and approach, we aim to inspire others, in and outside ICU walls, to make more efficient use of data at hand, now and in the future.

Virtual Reality to Improve Sequelae of the Postintensive Care Syndrome: A Multicenter, Randomized Controlled Feasibility Study.

Psychologic sequelae after critical illness, part of the postintensive care syndrome, significantly decrease quality of life. A robustly effective treatment intervention is currently lacking. Virtual reality has beneficial effects on several non-ICU-related psychologic disorders. The aim of this study was to explore patient-related determinants of ICU-specific virtual reality, such as the timing of patients' self-reported readiness to initiate virtual reality and the number of desired sessions and safety, and to explore the effects of ICU-specific virtual reality on mental health. DESIGN: A multicenter, randomized controlled feasibility study. SETTING: ICU at a university teaching hospital and a secondary care hospital in Rotterdam, The Netherlands. PATIENTS: Consecutive mechanically ventilated patients with sepsis or septic shock. INTERVENTIONS: Patients were randomly assigned (1:1) to receive ICU-specific virtual reality (ICU-specific virtual reality group) or exposure to a nature virtual reality environment (control virtual reality group). MEASUREMENT AND MAIN RESULTS: Explorative outcomes were feasibility, in terms of patient-related determinants, and safety. The effects of ICU-specific virtual reality on the psychologic components of postintensive care syndrome and quality of life were additionally studied. Fifty patients (median age: 61 yr; 21 [42%] female) were included. Patients in the ICU-specific virtual reality group felt ready to initiate the virtual reality intervention 10 days (median, 95% range, 5-21 d) after ICU discharge, and one session (median, 95% range, 1-6) was desired. ICU-specific virtual reality patients experienced higher immersion, cybersickness scores were low, and no changes in vital signs were observed. They also reported reduced posttraumatic stress disorder and depression scores and better mental health from 2 days until 1 month after initial exposure (Short Form-12 Mental Component Scale: ICU-specific virtual reality, 57 [36-67] vs control virtual reality, 47 [26-63]; p < 0.01). Six months after exposure, this effect was still present for posttraumatic stress disorder and depression, but not for mental quality of life. CONCLUSIONS: ICU-specific virtual reality is a feasible and acceptable novel intervention that could be used during recovery from an episode of critical illness in the ICU. A future, adequately powered study should confirm whether virtual reality is able to improve mental health and quality of life.